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An efficient joint source-channel (s/c) decoder based on the side information of the 
source and on the MN-Gallager algorithm over Galois fields is presented. The dynamical 
block priors (DBP) are derived either from a statistical mechanical approach via calculation 
of the entropy for the correlated sequences, or from the Markovian transition matrix. The 
Markovian joint s/c decoder has many advantages over the statistical mechanical approach. 
In particular, there is no need for the construction and the diagonalization of a g x g matrix 
and for a solution to saddle point equations in q dimensions. Using parametric estimation, 
an efficient joint s/c decoder with the lack of side information is discussed. Besides the 
variant joint s/c decoders presented, we also show that the available sets of autocorrelations 
consist of a convex volume, and its structure can be found using the Simplex algorithm. 

§1. Introduction 

Source coding is a process for removing redundant information from the source 
information symbol stream. Channel coding is a procedure for adding redundancy 
as protection into the information stream which is to be transmitted. During the 
last decade engineers and also physicists have designed efficient error correction tech- 
niques such as Low-Density-Parity-Check-Codes (LDPC)1),2) or Turbo codes, that 
nearly saturate Shannon's limit. 

The Shannon separation theorem states that source coding (compression) and 
channel coding (error protection) can be performed separately and sequentially, while 
maintaining optimality 1),3). However, this is true only in the case of asymptoti- 
cally long block lengths of data and point-to-point transmission. In many practical 
applications, the conditions of the Shannon's separation theorem neither hold, even 
approximatly. Thus, considerable interest has developed in various schemes of joint 
source-channel (s/c) coding, where compression and error correction are combined 
into one mechanism. 

The paper is organized as follows. In Section 2, Statistical Mechanical (SM) 
methods are used to explore properties of correlated sequences. In section 3, the 
space of possible sets of autocorrelations is investigated. In sections 4 Mackay and 
Neal's (MN) algorithm for error correction is briefly introduced, and the results of 
section 2 are used to extend this algorithm to a joint s/c scheme. The estimation 
of the threshold of the code using the scaling argument for the convergence time is 
presented in section 5. Section 6 presents simulation results, and compares joint to 
separation schemes. In Sections 7 and 8, the joint s/c problem is attacked from a 
different point of view, a Markovian approach, and an efficient joint scheme with the 
lack of side information is demonstrated. 
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§2. Joint s/c coding - Statistical Mechanical approach 

In our recent papers4),5) a particular scheme based on a SM approach for the 
implementation of the joint s/c coding was presented and the main steps are briefly 
summarized below. The original boolean source is first mapped to a binary source 
6) {xi = ±1} i = I,...,L, and is characterized by a finite set of autocorrelations 
bounded by the length ko 



m 

J^XiY[x(^i+k,) mod L (2-1) 



L 

i=l j=0 



where km < k^ is the highest length autocorrelation taken and the total number of 
possible different autocorrelations is 2*''°. For /cq = 2, for instance, there are only 4 
possible correlations, Co, Ci, C2 and C12, and for ko = 3 there are 8 possible different 
correlations; Cq, Ci ,€2, C3, C12, C13, C23, C123, where we do not assume left-right 
symmetry for the source. Note that for general kQ and m = 1, Eq. ()2-l|) reduces to 
the two-point autocorrelation function 7). The number of sequences obeying these 
2^0 constraints is given by 

L m 

^ = rr|^^=±i} Xi JJ Xi+fc^. - LCki^...,kJ (2-2) 

{fcl,fc2,.--,fcm} i = l j = 

where m = stands for Cq. Using the integral representation of the delta functions, 
Eq. (|2-2|) can be written as 



„ m 

W \\(^y{ki,..M^^v(^-yk^,..,kraCki,..,kjTrey.^^ 

{ki,..,km} ki,..,km i j=0 

Since kj < k^, the last term of this equation indicates that the trace can be performed 
using the standard transfer matrix (of size 2^^° x 2*^°) methods). More precisely, 
assume two successive blocks of ko binary variables denoted by (xi,...,Xfco) and 
(xkQ+i, ...,X2ko)- The element of the transfer matrix is equal to the value of the 
last exponential term (on the r.h.s of the trace), where the first block is in state i 
(among 2^" possible states) and the second block is in state j. The transfer matrix 
is a non-negative matrix (as long as the yki....,kr„ '^^al numbers), and the leading 
eigenvalue is positive and non-degenerate8). In the leading order one finds 

^ = j dykexp{-L[^yk^,...,krr^Ckl,...,k,^ - lnAmax({yfci,...,fc„})]} (2-3) 

where Xmax is the maximal eigenvalue of the corresponding transfer matrix. For 
large L and using the saddle point method, the entropy, H2{{Cki,...,km})j is given in 
the leading order by 

11 

-f^2 ({Cfc,...,fc„}) = — [— InAmax ({yfci,...,fc™}) - ^ yki,...,kmCki,...,kJ\ (2-4) 
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where {yki,...,km} ^'^^ determined from the saddle point equations of Q 4), 5). As- 
suming a Binary Symmetric Channel (BSC) and using Shannon's lower bound, the 
channel capacity of sequences with a given set of autocorrelations bounded by a 
distance /cq is given by 

C- ^-H2{f) 

where / is the channel bit error rate and pf, is a bit error rate. The saddle point 
solutions derived from Eq. (|2-4p indicate that the equilibrium properties of the one- 
dimensional Ising spin system (xj = ±1) with up to order ko multi-spin inter actions9) 

fco m 

H = -EE'-^-^U-^^^. (2-6) 

i k=l ^ j=0 

obey in the leading order the autocorrelation constraints, Eq. (|2-lj) . This property of 
the effective Hamiltonian, Eq. H2-6() . is used in simulations to generate an ensemble 
of signals (source messages) with the desired set of autocorrelations. Note that in 
the following we choose (5 = 1, and hence we denote {yki,...,km} '^^ interactions. 

§3. Possible sets of autocorrelations and the Simplex algorithm 

The entropy of correlated sequences can be calculated from Eq. H2-4|) . For the 
simplest case of sequences obeying only Ci and C2 the numerical solution of the 
saddle point equations indicate that the entropy is positive only in the regime 

-(l + C2)/2 < Ci < (l + C2)/2 (3-1) 

where outside of this regime the entropy is zero. At the boundaries, Ci = |(1-|-C2)/2|, 
two phenomena are observed: (a) the entropy falls abruptly to zero at the boundary, 
and (b) yi and —y2 diverge at the boundary (the one-dimensional Hamiltonian, Eq. 
H2-6() consists of frustrated loops). 

These limited results obtained from the numerical solutions of the saddle point 
equations suffer from the following limitations: (a) finding the boundaries of the 
region in the space of {Cki^...^km} with a finite entropy is very sensitive to the numer- 
ical precision since on the boundary the {|yi|} diverge; (b) it is unclear whether the 
available space consists of a connected regime; (c) the question of whether out of the 
space with a finite entropy, there are a finite or infinite number of sequences (for in- 
stance e^) obeying the set of autocorrelations cannot be answered using the saddle 
point method; (d) extension of the saddle point solutions to identify the boundaries 
of the finite entropy regime to many dimensions is a very heavy numerical task. 

To overcome these difficulties, we show below how the possible sets of autocor- 
relations can be identified using the Simplex algorithm. 

For the case of only two constraints Ci and C2, for instance, following the 
methodology of the transfer matrix, let us concentrate on four successive binary 
variables Si, Si+i, Si+2, Si+3, where Si = ±1. Since the Hamiltonian, Eq. ()2-6|) . 
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obeys in this case an inversion symmetry, let us examine only the 8 configurations 
out of 16 where 5*3 = — , (it it —it). For these 8 configurations one can assign 
the following marginal probabilities, P±±-±, where each probability stands for the 
fraction of sequences obeying Ci and C2 with a given state for these four successive 
binary variables. In the SM language we measure the probabilities of these four states 
in thermal equilibrium of the micro-canonical ensemble obeying Eq. (|2-H) . It is clear 
that the Hamiltonian, Eq. (|2-H|) . is translationally invariant, P{Si, Si-\-i, Si+2, Si+3) 
is independent on i after averaging over all sequences obeying constraints (|2-1() . 
For these 8 marginal probabilities one can write the following 14 equations: 
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0<P±±-±<1 (3-2) 

For a given Ci, these 14 equations can be solved for the minimum and the maximum 
available C2 using the Simplex method. Running over values of —1 < Ci < 1, 
we indeed recover the result of Eq. H3-l|) . However, the Simplex solution indicates 
the lack of even a finite number of sequences beyond the regime with finite entropy. 
Hence, a simple geometrical calculation obeying constraint 1311 indicates that the 
fraction of the 2D space (-1:1,-1:1) of (Ci,C2) with available sequences is 1/2. 

For the case of three constraints, Ci,C2 and C3, one can similarly write 45 
equalities and inequalities for the 32 probabilities of 6 successive binary variables 
P±±±±-±- For a given Ci and C2, these 45 equations and inequalities can be solved 
for the minimum and the maximum available C3 using the Simplex method. The 
Simplex solution indicates: (a) the solution space in the three-dimensional box (— 1 : 
1, —1 : 1, — 1 : 1) for (Ci, C2, C3) is a connected region bounded by a few planes. This 
result is consistent with the solution obtained from Eqs. H2-3l2-4() : (b) the fraction 
of the volume of the box obeying the three constants is ~ 0.222. Preliminary results 
indicate that for 4 (Cj, i = 1,2,3,4) and 5 (Cj, i = 1,2,3,4,5) constraints the 
available volume is ~ 0.085, 0.034, respectively. 

The fraction of possible sets of autocorrelations appears to decrease as the num- 
ber of constraints increases. However, the question of whether the fraction of avail- 
able autocorrelations drops exponentially with the number of constraints as well as 
its detailed spatial shape is the subject of our current research. 

We conclude the discussion in this section with the following general result. The 
available volume for the general case of q constraints {Cki,...,km}i < log2(9), is 
convex. The main idea is that one can verify that the set of equalities can be written 
in a matrix representation in the following form 

MP = C (3-3) 
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where M is a matrix with elements ±1; P represents the marginal probabilities 
P(±, lb, ....) and C represents the desired correlations or a normalization constant 
(for instance Ci/2, C2/2 and 1/2, for the case of Eq. ()3-2() ). The inequalities force 
the probabilities into the range [0 : 1]. Clearly if Pi(ib, it, ...) and P2{^,^, ■■■) are 
two sets of probabilities obeying Eq. (|3-3() then 

APi + (1 - A)P2 (3-4) 

is also a solution of the set of the equalities (0 < A < 1). Hence, the available volume 
is convex. 

§4. Joint s/c decoder: Statistical Mechanical approach 

The transfer matrix method indicates that the relevant scale of the correlated 
source message is /cq. Hence, our encoding/decoding procedure is based on the MN 
code for a finite field q = 2^^ 10), which is based on the construction of two sparse 
matrices A and B of dimensionalities {Lq/R)xLq and {Lq / R) x (Lq / R) respectively, 
where R is the code-rate and the number of symbols in the source is Lq = L/kQ. 
The matrix B~^A is then used for encoding the message 

t = B-^Ax {mod q) (4-1) 

The finite field message vector t is mapped to a binary vector and then transmitted. 
The received message, r, is corrupted by the channel bit error rate, /. 

The decoding of symbols of k^ successive bits (named in the following as a block 
of bits or binary variables) is based on the solution of the syndrome 

Z = Br = Ax + Bn { mod q) (4-2) 

where n stands for the corresponding noise of ko successive bits. The solution of 
the Lq/ R equations with Lq[1/R + 1) variables is based on the standard message 
passing algorithm introduced for the MN decoder over Galois fields with q = 2^nQ) 
and with the following modification. The horizontal pass is left unchanged, hut a 
dynamical set of probabilities assigned for each block is used in the vertical pass. The 
Dynamical Block Probabilities (DBP), {P^}, are determined following the current 
belief regarding the neighboring blocks and are given by 

I'n = Si (c) ( ^ q[SL (/, c) ] ( ^ q^^Sn (c, r) ] ; P„= = — ^ (4-3) 

\l=l J \r=l J Z^j=l7n 

where l/r/c denotes the state of the left/right /center (n— 1 /n+1 / n) block respec- 
tively and qi/q^p> are their posterior probabilities. Si{c) = e~^^^ stands for the Gibbs 
factor of the inner energy of a block (ko successive binary variables spins), character- 
ized by an energy Hj at a state c, see Eq. H2-6|) . Similarly Sl{1,c) (5'ij(c, r)) stands 
for the Gibbs factor of consecutive Left/Center (Center /Right) blocks at a state l,c 
(c, r) 4), 5). The complexity of the calculation of the block prior probabilities is 
0{Lq^ / \ogq) where L/logg is the number of blocks. The decoder complexity per 
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iteration of the MN codes over a finite field q can be reduced to order 0{Lqu)ll), 

total com- 




o 

■ Checks. 

• Source variable. O- 
O Noise variable. q: 
^TM check 



Fig. 1. A message passing in the joint s/c de- 
coder is represented by a bipartite graph 
with an additional layer. 



1 bit per 
row/column 
3 bit per 
row/column 
3 bit per 
row/column 



B' 

U/R 




Fig. 2. The structure of the matrices A and 
B for the MN decoder taken from refer- 
encel2), for rate 1/3. The black dots (area) 
denote the non-zero elements of the matri- 
ces A, B-^. 



Another way to represent the dynamical behavior of the SM joint s/c decoder is 
in the framework of message passing on a graph (Fig. Typically for LDPC, the 
graph is bipartite and consists of variable nodes (circles) and check nodes (squares). 
For the MN algorithm, there are two types of variable nodes: source nodes (filled 
circles) and noise nodes (open circles). A message from variables to checks is a 
horizontal pass, and a message from checks to variables is a vertical pass. In the 
joint s/c decoder there is a third layer (diamonds): each element in this layer sends a 
message (outbound arrow) to a single source variable (namely, the dynamical block 
prior), and receives two messages (inbound arrows) from the neighboring source 
variables (namely their a-posteriori probabilities). 

For simplification of the discussion below, in almost all of the simulation results 
we concentrate on rate 1/3 and the construction of the matrices A and B follow 
reference 12) which is sketched in Fig. |21 The advantage of this construction is that 
the matrices A and B are very sparse, but the threshold of the code for large blocks is 
only 1 — 3% of the channel capacityl2), 13). Furthermore, since B has a systematic 
structure, the complexity of the encoder scales linearly with L although B~^ is 
densel4), 15). Of course, codes with higher thresholds exist, hence the performance 
of the joint s/c algorithm reported below should be interpreted as a lower bound. 
(Results for a limited example with rate greater than one, R> 1, are briefly discussed 
in reference 16)) 

We conclude this section with the comment that the possibility of the SM joint 
s/c algorithm in the framework of the MN-Gallager decoder to the Gallager decoder2) 
is in question. In the Gallager decoder we first solve Lq{1/R — 1) equations for 
the noise variables, and only in the final step is the message recovered. Since the 
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noise is not spatially correlated, we do not see a simple way to incorporate in the 
Gallager case the side information about the spatial correlations among the message 
variables. The equivalence between these two (MN-Gallager and Gallager) similar 
decoders therefor also in doubt. 

For illustration, in Fig. El we present results for rate i? = 1/3, L = 10, 000, (7 = 4 
and 8 where the decoding is based on the dynamical block posterior probabilities, Eq. 
(|4-:-{|) . and with the following parameters. For q = A (open circles) Ci = 0.55, C2 = 
0.5, C12 = 0.4 {yi = 0.275, y2 = 0.291, yi2 = 0.149) and H2 = 0.683. Shannon's 
lower bound, Eq. H2-5|) . is denoted by the double dotted line, where for = the 
channel noise level is /c = 0.227. For g = 8 (open diamonds) Ci = 0.77, C2 = 
0.69, = 0.56, C7i23 = 0.7 (yi = 0.349, 7/2 = 0.36, 7/3 = -0.211, 7/123 = 0.443) 
and H2 = 0.453. Shannon's lower bound is denoted by the dashed line, where for 
Pft = the channel noise level is /c = 0.275. Each point was averaged over at least 
1, 000 messages. These results for both q = 4 and 8 indicate that the threshold of the 
presented decoder with L = 10, 000 is ~ 15% — 20% below the channel capacity for 
infinite source messages. It is worth mentioning, that without using the dynamical 
block priors, Eq. ()4-3() . the decoder fails to decode at /c =~ 0.13 — 0.14. 













cj-8 bound 


° :4 


/ ! 

/ 




11=4 

■ — ■ q-4 tiound 


' 



/ 
/ 




, , , , 1 , 


1 





0.1 0.15 0.2 0.25 



/ 

Fig. 3. Simulation results for rate R = 1/3, 
L — 10, 000, q = 4 and 8. Each point was 
averaged over at least 1,000 source mes- 
sages with the desired set of autocorrela- 
tions, (refer to text for description) 
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Fig. 4. The flip rate / as a function of 1/tmed 
for GF{4:) with Ci = C2 = 0.8 and L = 
1,000, 5,000 ,50,000. The lines are a re- 
sult of a linear fit. The threshold, /oo ~ 
0.272, extrapolated from the scaling behav- 
ior Eq. H5-l|l . is independent of L. 



§5. The threshold of the code 

The threshold fc for L ^ 00 is estimated from the scaling argument of the 
convergence time, which was previously observed for q = 212), 13). The median 
convergence time, measured in iterations of the MN algorithm, is assumed to diverge 
as the level of noise approaches fc from below. More precisely, we found that the 
scaling for the divergence of tmed is independent of q and is consistent with: 

_ A 

Jc ~ J 
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where for a given set of autocorrelations and q, A is a constant. Moreover, for a 
given set of autocorrelations and a finite field q, the extrapolated threshold fc is 
independent of L, as demonstrated in Fig. ^ This observation is essential to de- 
termine the threshold of a code based on the above scaling behavior. Note that the 
estimation of tmed is a simple computational task in comparison with the estimation 
of low bit error probabilities for large L, especially close to the threshold. We also 
note that the analysis is based on tmed instead of the average number of iterations, 
ta^,12) since we wish to prevent the dramatic effect of a small fraction of finite sam- 
ples with slow convergence or no convergence. 17) We note that preliminary results 
indicates that for a given {Ck}, fc{Q) appears asymptotically to be consistent with 
fc{q) '-^ fc- const /q. 

§6. Comparison between joint and separation schemes 

Results of simulations for q = 4, 8, 16 and 32 and selected sets of autocor- 
relations are summarized in Table I (Fig. [5J and the definition of the symbols is: 
{Ck} denotes the imposed values of two-point autocorrelations as defined in eqs. 1211 
and 12-21 {yk} are the interactions strengths, Eq. ()2-6|) : H represents the entropy 
of sequences with the given set of autocorrelations, Eq. (|2-4() : fc is the estimated 
threshold of the MN decoder with the DBF derived from the scaling behavior of 
tmed, Eq. (|5-1() : fsh is Shannon's lower bound, Eq. ()2-5() : Ratio is the efficiency of 
our code, fc/ fsh] '^R indicates the gzip compression rate averaged over files of the 
sizes 10^ — 10^ bits with the desired set of autocorrelations. We assume that the 
compression rate with L = 10® achieves its asymptotic ratio, as was indeed confirmed 
in the compression of files with different L; XjB* indicates the ideal (minimal) ratio 
between the transmitted message and the source signal after implementing the fol- 
lowing two steps: compression of the file using gzip and then using an ideal optimal 
encoder /decoder, for a given BSC with fc- A number greater than (less than) 3 in 
this column indicates that the MN joint s/c decoder is more efficient (less efficient) 
in comparison to the channel separation method using the standard gzip compres- 
sion. The last four columns of Table I (Fig. [HJ are devoted to the comparison of 
the presented joint s/c decoder with advanced compression methods. PPMr and 
ACr represent the compression rate of files of the size 10^ — 10® bits with the desired 
autocorrelations using the Prediction by Partial MatchlS) and for the Arithmetic 
Coder 19), respectively. Similarly to the gzip case, 1/Rppm and 1/Rac denote the 
optimal (minimal) rate required for the separation process (first a compression and 
then an ideal optimal encoder/decoder) assuming a BSC with fc- 

Table I indicates the following main results: (a) a degradation in the performance 
is observed as the correlations are enhanced, (b) The efficiency of our joint s/c coding 
technique is superior to the alternative standard gzip compression in the separation 
technique. The gain disappears as the entropy decreases. These results are farther 
discussed in 5) . 

The DBF decoder based on the SM approach suffers one major disadvantage: 
computing the interactions, which involves finding the eigenvalues of aqxq matrix 
(0(g^)), and solving the saddle-point equations. 12-41 which is also a heavy numerical 
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Fig. 5. Results for g = 4, 8, 16, 32 and selected sets of two-point autocorrelations {Ck} 
task, even for > 4. 

§7. Markovian joint s/c decoder 

In order to overcome the abovementioned drawbacks of the SM approach, we 
now treat the source sequence from a different point of view, by assuming that 
these sequences were generated by a Markov process. Hence, the sequence can be 
described by an alphabet GF{q), a transition matrix T^j = P{xi = = /c), 

and the stationary solutions of the process, P{j)- The key point is that for large 
messages, T and P{j) can be (approximately) measured by the sender (for each 
chunk of data) with 0{Lq) operations. T and P{i) may then be transmitted reliably 
to the decoder and used as side information in the decoding process (similar to the 
transmission of the yi, ■■■yko interactions in the SM approach). 

Consider three successive symbols Xj, Xj+i in such a sequence. The proba- 
bility of the triplet a, b, c is given by: 

P{a, b, c) = P{a, b) ■ P(c|a, b) = P{a, b) ■ P(c|6) = ^("'^|^^(^' ^) (7.1) 

where use has been made of the Bayes Rule: P{x,y) = P{x) ■ P{y\x), and fact that 
the process is memoryless. Now, given the a-posteriori probabilities for the first and 
last symbols in the triplet: qf_i = Pr{xi-i = a) and qf^-^ = Pr(xi+i = c), one 
can calculate a dynamical block prior (corresponding to the prior in ()4-3() for the 
probability that Xi = b: 



Pr{xi = 6) = 1 . ^ P{a, b, c) ■ q{a) ■ q{c) = 

a,c=l 




where Z is a normalization constant such that: Ylb=i ^^i^i = b) = 1. 

The extension of the MN algorithm to the joint source-channel case consists of 
the following steps: 
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1. A binary sequence of Lq ■ log2{q) bits is converted to Lq GF{q) symbols. 

2. The encoder measures T and -P(j) for all the q symbols over the source, and 
transmits reliably this side information to the decoder. 

3. The source is encoded according to (|4-1() . then reconverted to binary represen- 
tation and transmitted. 

4. The decoder maps the received signal back to GF{q), and performs the regular 
decoding (|4-2|) , but after every iteration of message passing, the prior for each 
source symbol is recalculated according to ()7-2() . 

The complexity of calculating the q priors for a single symbol according to the 
posteriors of its neighbors is reduced from q^ to q^ by Eq. (|7-2|) . hence the decoder's 
complexity remains linear, with total complexity of 0{LQqu + L^q^) per iteration. 

However, there is still a need for the 
transmission of the side information (T, 
P{j))- Hence the size of the header is 
of the order of 0{q^). For L — > oo or 
more precisely for L ^ q^ the overhead 
of the transmitted side information is 
negligible; however, for a finite L < q^ 
it may cancel the benefits of the Marko- 
vian joint s/c decoder. 

The header size can be reduced 
using the following observation: For 
sequences with enhanced autocorrela- 
tions, the structure of T was observed to 
be dominated by a small number of large 
elements. One can transmit only these 
dominated elements, and the remaining 
elements of each row of T will be filled 
evenly to maintain: Yl'j=i'^ij ~ ^■ 

Fig. El represents simulation results 
for the Markovian scheme. Triangles 
mark decoding using full T, while circles 
mark decoding using only the 8 (out of 
64) most dominant elements of T. The 
performance seems to be only slightly af- 
fected by this approximation, which dra- 
matically reduces the required transmitted side information. 

An interesting open question is the effect of use of the sparseness of T on the 
complexity and performance of the decoder. 

§8. Efficient Joint s/c decoder with the lack of side information 

The discussion in the previous sections indicates that the performance of the 
presented joint s/c coding is not too far from Shannon's lower bound and, most 
probably, using an optimized code (a better construction for the matrices A and B 



10 



10" 



10 



10' 



10- 



A 
O 



Q Markov (8 elements) 
A Markov 



OA 



i 



0.22 0.24 0.26 0.28 

/ 



Fig. 6. The bit error rate, pb versus the chan- 
nel bit error rate / for L = 10, 000, R = 
1/3, g = 8 with Ci = C2 = Cs = 0.7. 
Decoding following the Markovian process, 
Eq. 17-21 1 (open triangle), decoding follow- 
ing the Markovian process where only 8 
dominated elements of the transition ma- 
trix, T, are taken as a side information 
(open circle) . Shannon's lower bound, fc = 
0.271 {H = 0.47), is denoted by an arrow. 
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of the MN code), the channel capacity can be nearly saturated. However, for a finite 
block length, the main drawback of our algorithm is the overhead of the header 
which must be encoded and transmitted reliably. One has to remember that the 
size of the header scales with where the precision of each element is of the order 
0(log L). This overhead is especially intolerable in the limit where log /L ~ 0(1) 
Note that this is indeed the situation even for very large messages, L = 10^, and the 
largest taken autocorrelation length is only log2 g = 8. 

In this section we explain how the Markovian joint s/c can be implemented 
without any side information. The key points are the special properties of KS con- 
struction 12) (Fig. 121): the first Lq rows of A are characterized by one non-zero 
element per row and column, where the first Lq rows of B are characterized by 2 
non-zero elements. Furthermore, due to the systematic form of B, each row cannot 
be written as a linear combination of the other rows. Hence, the first Lq bits of the 
syndrome, Eq. (|4-2|) . are equal (up to a simple permutation) to the source, with an 
effective flip rate, feff- For GF{2) for instance, Zj = Xi + nj + n^+i (i marks the 
position of the nonzero element in the j^^ row of A), and feff = 2/(1 — /). The 
first Lq symbols of Z are therefore a result of a Hidden Markov Model (HMM). The 
underlaying transition matrix, T, generating the source sequence, can be estimated 
by means of the EM algorithm 20). This is a standard tool for solving such Paramet- 
ric Estimation problems, which has linear complexity. Having T (approximately) 
revealed, the Dynamical Block Priors can be used as described in 17-21 

For the general construction of the MN algorithm, one adds/subtracts rows of 
the concatenated matrix [A, and the corresponding symbols in z (see Eq. (j4-2j) ). 
such that the following situation is finally reached: The first Lq rows of A are the 
identity matrix, regardless of the construction of the first Lq rows of B. From the 
knowledge of the noise level / and the structure of i^^ row of B one can now calculate 
the effective noise level, fi,eff, of the i*^ received source symbol. Since all {/i,e//} 
are functions of a unique noise level /, one can again estimate the parameters of 
the Markovian process using some variants of the EM algorithm. Note, that in the 
general case the first Lq rows of B contain loops, hence the {/i,e//}) are correlated. 
However, these correlations are assumed to be small as the typical loop size is of the 
order of 0(log(L))21). 

§9. Concluding remarks 

The only remaining major drawback of the presented Markovian joint s/c coding 
is that the complexity of the decoder per message passing, scales as 0(-Lg^/ log2 (<?)), 
this may considerably slow down the decoder even for moderate alphabet size. Note 
however, that for large q, such that > L, and low entropy sequences, the transition 
matrix, T, is expected to be very sparse (consider q = 1024 vs. block size of L = 
100,000). Taking advantage of the sparseness of T, the complexity of the decoder 
can be further reduced. 

The one-dimensional Markovian joint s/c decoder can be easily extended to 
coding of a two-dimensional array of symbols or even to an array of symbols in 
higher dimensions. The complexity of the DBF decoder scales as L'^q'^'^'^^, where 
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Lq is the number of blocks in the array, and d denotes the dimension22). Using 
Markovian and Bayesian assumptions, the complexity can be reduced to 0{LQq'^). 
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